OpenACC port of halo exchanges #1355
Merged
mgduda merged 2 commits intoMPAS-Dev:developfrom Mar 20, 2026
Merged
Conversation
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
2 times, most recently
from
56ed028 to
a8fda92
Compare
mgduda
reviewed
Sep 24, 2025
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
d0c1431 to
1e08917
Compare
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
1e08917 to
eff87cf
Compare
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
eff87cf to
3d2c334
Compare
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
3d2c334 to
5486740
Compare
abishekg7
changed the title
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
5486740 to
6ce8e33
Compare
mgduda
requested changes
Mar 17, 2026
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
6ce8e33 to
2ae3105
Compare
mgduda
requested changes
Mar 19, 2026
mgduda
approved these changes
Mar 19, 2026
This commit enables execution of halo exchanges on GPUs via OpenACC directives, if the MPAS atmosphere core has been built with an appropriate GPU-aware MPI distribution. Module mpas_halo is modified in the following ways to enable GPU-aware halo exchanges: - In the call to mpas_halo_exch_group_complete, OpenACC directives copy to device all the relevant fields and metadata that are required for the packing and unpacking loops later. - OpenACC directives are introduced around the packing and unpacking loops to perform the field to/from send/recv buffer operations on the device. The attach clauses introduced to the parallel constructs ensures that the device pointers are attached to the device targets at the start of the parallel region and detached at the end of the region. - The actual MPI_Isend and MPI_Irecv operations use GPU-aware MPI, by wrapping these calls within !$acc host_data constructs. Note: This commit introduces temporary host-device data movements in the atm_core_init routine around the two calls to exchange_halo_group. This is required just for this commit as all halo-exchanges occur on the device and fields not yet present on the device must be copied over to it before the halo exchanges and back to host after it. These copies will be removed in subsequent commits.
Introducing a new namelist option under development, config_gpu_aware_mpi, which will control whether the MPAS runs on GPUs will use GPU-aware MPI or perform a device<->host update of variables around the call to a purely CPU-based halo exchange. Note: This feature is not available to use when config_halo_exch_method is set to 'mpas_dmpar'
abishekg7
force-pushed
the
framework/acc_halo_exch
branch
from
d52ce31 to
b0ce630
Compare
mgduda
added
Atmosphere
Framework
OpenACC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enables execution of halo exchanges on GPUs via OpenACC directives, when the MPAS atmosphere core has been built with an appropriate GPU-aware MPI library. This PR builds on the OpenACC memory consolidation introduced in #1315.
Module
mpas_halois modified in the following ways to enable GPU-aware halo exchanges:mpas_halo_exch_group_complete, OpenACC directives copy to device all the relevant fields and metadata that are required for the packing and unpacking loops later.attachclauses introduced to the parallel constructs ensures that the device pointers are attached to the device targets at the start of the parallel region and detached at the end of the region.MPI_IsendandMPI_Irecvoperations use GPU-aware MPI, by wrapping these calls withinIn addition, this PR introduces a new namelist option under the
developmentgroup,config_gpu_aware_mpi. When set totrue, it switches on GPU-direct halo exchanges. The default setting isfalse, in which halo exchanges occur on the host, which necessitates host<->device data transfers around the halo exchanges as necessary.Note: This feature is not available to use when config_halo_exch_method is set to 'mpas_dmpar'